NPS-AM-12-C9P22R02-088 


EXCERPT  FROM  THE 


PROCEEDINGS 

OF  THE 

NINTH  ANNUAL  ACQUISITION 
RESEARCH  SYMPOSIUM 
THURSDAY  SESSIONS 
VOLUME  II 


Affordability  Engineering  Framework  Overview 

Scott  Anderson,  Virginia  Wydler,  and  Joe  Duquette 
The  MITRE  Corporation 

Published  April  30,  2012 


Approved  for  public  release;  distribution  is  unlimited. 
Prepared  for  the  Naval  Postgraduate  School,  Monterey,  CA  93943. 


Form  Approved 
OMB  No.  0704-0188 


Report  Documentation  Page 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 


1.  REPORT  DATE 

30  APR  2012 

2.  REPORT  TYPE 

3.  DATES  COVERED 

00-00-2012  to  00-00-2012 

4.  TITLE  AND  SUBTITLE 

Affordability  Engineering  Framework  Overview 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

The  MITRE  Corporation, 202  Burlington  Road, Bedford, MA, 01730-1420 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS (ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

The  current  economic  environment  and  mounting  federal  budget  deficits  are  placing  considerable 
economic  stress  on  the  Department  of  Defense  (DoD)  and  other  government  agencies.  Investments  for  new 
capabilities,  upgrades,  and  enhancements  to  existing  systems  as  well  as  simple  continuations  of  existing 
programs  require  careful  analysis  and  evaluation  of  their  affordability,  efficiency,  and  effectiveness.  The 
Affordability  Engineering  Framework  (AEF)  is  being  designed  to  help  the  DoD  respond  to  these  imminent 
fiscal  realities  and  advance  the  practice  of  affordability  engineering  to  improve  acquisition  program 
success.  The  AEF  is  a  structured,  actionable  approach  with  tools  and  techniques  to  address  affordability 
challenges  throughout  the  life  cycle.  The  AEF  uses  multi-disciplinary  teams  to  quantitatively  evaluate 
program  affordability  while  identifying  integrated  cost,  schedule,  and  performance  trade  space.  The  AEF 
includes  four  steps:  an  affordability  risk  assessment,  a  validation  approach  for  coupling  technical  baselines 
and  program  cost  estimates,  a  deliberate  tradeoff  process,  and  the  generation  of  preferred  courses  of  action 
with  a  recommendation  based  on  a  portfolio  analysis  methodology.  The  AEF  can  provide  benefits  across  a 
wide  range  of  acquisition  programs  and  provide  the  affordability  information  for  data-driven  program 
decision-making.  In  the  coming  months,  the  AEF  will  be  piloted  and  migrated  across  selected  DoD 
programs  for  implementation  with  iterative  evaluation  and  development. 


15.  SUBJECT  TERMS 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 
ABSTRACT 

18.  NUMBER 
OF  PAGES 

19a.  NAME  OF 
RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Same  as 
Report  (SAR) 

38 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


The  research  presented  at  the  symposium  was  supported  by  the  acquisition  chair  of  the 
Graduate  School  of  Business  &  Public  Policy  at  the  Naval  Postgraduate  School. 

To  request  defense  acquisition  research  or  to  become  a  research  sponsor,  please 
contact: 

NPS  Acquisition  Research  Program 
Attn:  James  B.  Greene,  RADM,  USN,  (Ret.) 

Acquisition  Chair 

Graduate  School  of  Business  and  Public  Policy 

Naval  Postgraduate  School 

Monterey,  CA  93943-5103 

Tel:  (831)656-2092 

Fax:  (831)656-2253 

E-mail:  ibqreene@nps.edu 

Copies  of  the  Acquisition  Research  Program’s  sponsored  research  reports  may  be  printed 
from  our  website  (www.acquisitionresearch.net). 


Preface  &  Acknowledgements 


Welcome  to  our  Ninth  Annual  Acquisition  Research  Symposium!  This  event  is  the 
highlight  of  the  year  for  the  Acquisition  Research  Program  (ARP)  here  at  the  Naval 
Postgraduate  School  (NPS)  because  it  showcases  the  findings  of  recently  completed 
research  projects — and  that  research  activity  has  been  prolific!  Since  the  ARP’s  founding  in 
2003,  over  800  original  research  reports  have  been  added  to  the  acquisition  body  of 
knowledge.  We  continue  to  add  to  that  library,  located  online  at 

www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  60  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  hope  this  symposium  will  spark  even  more  participation. 

We  encourage  you  to  be  active  participants  at  the  symposium.  Indeed,  active 
participation  has  been  the  hallmark  of  previous  symposia.  We  purposely  limit  attendance  to 
350  people  to  encourage  just  that.  In  addition,  this  forum  is  unique  in  its  effort  to  bring 
scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  Seldom  will  you  get  the  opportunity  to  interact  with  so 
many  top  DoD  acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both 
in  the  formal  panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals, 
breaks,  and  the  day-ending  socials.  Many  of  our  researchers  use  these  occasions  to 
establish  new  teaming  arrangements  for  future  research  work.  In  the  words  of  one  senior 
government  official,  “I  would  not  miss  this  symposium  for  the  world  as  it  is  the  best  forum 
I’ve  found  for  catching  up  on  acquisition  issues  and  learning  from  the  great  presenters.” 

We  expect  affordability  to  be  a  major  focus  at  this  year’s  event.  It  is  a  central  tenet  of 
the  DoD’s  Better  Buying  Power  initiatives,  and  budget  projections  indicate  it  will  continue  to 
be  important  as  the  nation  works  its  way  out  of  the  recession.  This  suggests  that  research 
with  a  focus  on  affordability  will  be  of  great  interest  to  the  DoD  leadership  in  the  year  to 
come.  Whether  you’re  a  practitioner  or  scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
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•  Commander,  Naval  Sea  Systems  Command 

•  Program  Executive  Officer,  Integrated  Warfare  Systems 

•  Army  Contracting  Command,  U.S.  Army  Materiel  Command 
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•  Office  of  Procurement  and  Assistance  Management  Headquarters, 
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Evaluation 
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Mary  Lacey — Lacey  is  the  deputy  assistant  secretary  of  the  Navy  for  Research,  Development,  Test, 
And  Evaluation  (RDT&E).  She  is  the  senior  civilian  and  serves  as  the  senior  advisor  to  the 
ASN(RD&A)  for  research,  development,  test,  and  evaluation,  and  system  engineering.  She  has 
oversight  responsibility  for  all  science  and  engineering,  test  and  evaluation,  modeling  and  simulation, 
chief  systems  engineering  policy,  practices,  and  processes  for  ASN(RD&A). 

Lacey  also  oversees  the  Department  of  the  Navy  (DoN)  chief  systems  engineering  position  and 
the  DoN  deputy  for  test  and  evaluation.  She  is  the  functional  acquisition  workforce  competency  leader 
for  systems  engineering,  and  she  is  responsible  for  the  long-term  stewardship  of  Naval  Laboratories 
and  Warfare  Centers,  where  most  of  the  Navy’s  RDT&E  capabilities  reside.  She  serves  as  a  liaison 
with  industry,  academia,  federally  funded  research  and  developments  centers  (FFRDCs),  UARCs, 
and  outside  agencies. 

Lacey  entered  the  senior  executive  service  in  1996  and  has  38  years  of  federal  service. 

Lacey  held  the  position  of  deputy  program  executive  for  Aegis  Ballistic  Missile  Defense  (BMD). 
She  served  as  the  civilian  executive  counterpart  to  the  program  executive  in  creating,  managing,  and 
overseeing  Aegis  BMD  policies,  practices,  organization,  and  mission  execution.  She  also  served  as 
acting  executive  director — the  senior  civilian  advisor  to  the  MDA  director. 

Lacey  served  as  National  Security  Personnel  Systems  (NSPS)  program  executive  officer  (PEO). 
She  was  appointed  by  the  NSPS  senior  executive,  deputy  Secretary  of  Defense  Gordon  England, 
and  led  the  comprehensive  policy  and  program  office  for  the  design  and  implementation  of  NSPS. 

Lacey  was  technical  director  of  the  Naval  Surface  Warfare  Center  (NSWC),  where  she  was 
responsible  for  a  business  of  $4.6  billion  and  over  16,000  employees.  Lacey  formerly  served  as  the 
director  of  NSWC,  Indian  Head  Division,  which  specialized  in  Energetics  and  weapons  systems.  She 
also  served  as  head  of  the  systems  research  and  technology  department  and  director  of  science  and 
technology  for  NSWC  Dahlgren  Division. 

Lacey  began  her  career  with  the  Department  of  the  Navy  in  1973  as  a  federal  junior  fellow 
working  for  the  Naval  Ordnance  Laboratory  in  underwater  shock  testing  and  evaluation,  advanced 
weapons  systems,  firefighting  technology,  and  nuclear  weapons  safety. 

Lacey  earned  a  Bachelor  of  Science  degree  in  mechanical  engineering  from  the  University  of 
Maryland,  where  she  also  completed  graduate  work  in  control  systems  and  explosives.  Lacey’s 
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awards  include  the  Presidential  Rank  Distinguished  and  Meritorious  Executive,  DoD  Distinguished 
Civilian  Service,  the  Navy  Distinguished  Public  Service,  Superior  Civilian  Service,  Women  in  Science 
and  Engineering  Lifetime  Achievement,  University  of  Maryland  Distinguished  Engineering  Alumna, 
and  the  Federal  Laboratory  Consortium  Laboratory  Director  of  the  Year.  Lacey  serves  on  the 
University  of  Maryland  School  of  Engineering  Board  of  Visitors,  the  Women  in  Engineering  Advisory 
Board,  and  the  International  Council  of  Systems  Engineering  Foundation  Board. 

Lacey  brings  to  her  position  a  wealth  of  experience  and  valuable  insight  into  civilian  workforce 
issues.  Throughout  her  career,  Lacey  has  been  actively  involved  in  engineering  workforce 
development.  She  continues  to  serve  as  a  mentor  and  advisor  to  engineering  professionals.  She  has 
proven  expertise  in  managing  large,  diverse  workforces  and  in  leading  and  sustaining 
transformational  change. 
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Affordability  Engineering  Framework  Overview1 


Scott  Anderson — CAPT  Anderson,  (USN,  Ret.),  is  currently  assigned  as  the  director  for  acquisition 
integration  at  the  MITRE  Corporation.  He  is  responsible  for  integrating  efforts  across  the  company  to 
further  the  objectives  of  affordability,  efficiency,  and  effectiveness  while  improving  acquisition 
outcomes  for  government  sponsors.  Prior  to  joining  MITRE  in  2010,  Anderson  served  26  years  in  the 
U.S.  Navy  as  a  P-3  patrol  plane  pilot,  Navy  test  pilot,  systems  engineer,  and  major  program  manager. 
He  attended  the  U.S.  Naval  Academy,  graduating  in  1983,  and  the  Naval  Postgraduate  School  from 
1997  to  1999,  earning  master’s  and  engineer’s  degrees  in  aeronautical  engineering.  He  flew  3,000+ 
hours  in  24  aircraft  types  and  was  qualified  DAWIA  Level  III  in  Test  and  Evaluation,  SPRDE,  and 
Program  Management. 

Virginia  Wydler — Wydler  has  more  than  25  years  of  experience  in  federal  acquisition  and 
contracting,  both  government  and  commercial.  She  is  a  former  federal  employee  and  Navy 
contracting  officer  for  major  acquisitions,  including  Harrier  Aircraft  AV-8B  the  Defense  Super-Mini 
Computer  Program.  She  has  also  held  positions  with  private  and  public  consulting  firms,  providing 
acquisition  and  contracting  capabilities  with  Booz  Allen  Hamilton  and  the  MITRE  Corporation.  She 
holds  an  MS  in  national  security  strategy  from  the  Industrial  College  of  the  Armed  Forces;  an  MS  in 
acquisition  and  contracting  from  the  Naval  Postgraduate  School;  and  a  BS  in  business  administration 
from  the  University  of  Maryland.  She  is  a  certified  professional  contracts  manager  (CPCM),  a  fellow, 
and  a  member  of  the  NCMA  Washington,  DC,  Chapter. 

Joe  Duquette — Duquette  has  over  30  years  of  experience  in  leadership  roles,  finance,  operations 
and  program  management.  He  is  currently  with  The  MITRE  Corporation  as  a  senior  principal  analyst 
providing  economic,  business,  and  decision  analysis  support  to  U.S.  Government  executives  and 
project  managers.  He  has  also  served  in  the  U.S.  Air  Force  as  a  program  director,  and  worked  for  the 
Raytheon  Company  prior  to  joining  MITRE.  He  holds  a  BA  in  mathematics  from  State  University  of 
New  York  and  an  MBA  from  the  University  of  Montana. 

Abstract 

The  current  economic  environment  and  mounting  federal  budget  deficits  are  placing 
considerable  economic  stress  on  the  Department  of  Defense  (DoD)  and  other  government 
agencies.  Investments  for  new  capabilities,  upgrades,  and  enhancements  to  existing  systems 
as  well  as  simple  continuations  of  existing  programs  require  careful  analysis  and  evaluation 
of  their  affordability,  efficiency,  and  effectiveness.  The  Affordability  Engineering  Framework 
(AEF)  is  being  designed  to  help  the  DoD  respond  to  these  imminent  fiscal  realities  and 
advance  the  practice  of  affordability  engineering  to  improve  acquisition  program  success. 

The  AEF  is  a  structured,  actionable  approach  with  tools  and  techniques  to  address 
affordability  challenges  throughout  the  life  cycle.  The  AEF  uses  multi-disciplinary  teams  to 
quantitatively  evaluate  program  affordability  while  identifying  integrated  cost,  schedule,  and 
performance  trade  space.  The  AEF  includes  four  steps:  an  affordability  risk  assessment,  a 
validation  approach  for  coupling  technical  baselines  and  program  cost  estimates,  a  deliberate 
tradeoff  process,  and  the  generation  of  preferred  courses  of  action  with  a  recommendation 
based  on  a  portfolio  analysis  methodology.  The  AEF  can  provide  benefits  across  a  wide 
range  of  acquisition  programs  and  provide  the  affordability  information  for  data-driven 
program  decision-making.  In  the  coming  months,  the  AEF  will  be  piloted  and  migrated  across 
selected  DoD  programs  for  implementation  with  iterative  evaluation  and  development. 

Background 

The  current  economic  environment  and  mounting  federal  budget  deficits  are  placing 
considerable  economic  stress  on  the  Department  of  Defense  (DoD)  and  other  government 


1  The  views,  opinions,  and/or  findings  contained  in  this  report  are  those  of  The  MITRE  Corporation  and  should 
not  be  construed  as  an  official  government  position,  policy,  or  decision,  unless  designated  by  other 
documentation.  Approved  for  Public  Release  12-1207.  Distribution  unlimited.  ©2012  The  MITRE  Corporation.  All 
rights  reserved. 
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agencies.  As  such,  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  (USD[AT&L])  “Better  Buying  Power”  memorandum  for  the  acquisition  community 
highlighted  restoring  “affordability”  as  a  key  objective,  which  has  since  been  instantiated  in 
policy  and  statute  (Carter,  2010).  Furthermore,  the  General  Accounting  Office  (GAO)  and 
other  sources  continue  to  report  the  many  DoD  and  agency  programs  that  are  experiencing 
budget  and  schedule  overruns.  As  a  result,  investments  for  new  capabilities,  upgrades,  and 
enhancements  to  existing  systems  and  simple  continuations  of  existing  programs  will  require 
careful  analysis  and  evaluation  of  their  affordability,  efficiency,  and  effectiveness.  Budget 
reductions  are  mandating  difficult  decisions  about  where  to  invest  limited  resources,  how  to 
make  current  programs  more  affordable,  and  whether  to  terminate  poorly  performing 
programs.  There  is  a  need  for  the  DoD  to  respond  to  the  imminent  fiscal  realities  and 
advance  the  practice  of  affordability  engineering  for  long-term  acquisition  improvement.  A 
proposed  approach  to  achieving  affordability  through  a  robust  practice  of  affordability 
engineering  is  through  the  application  of  the  Affordability  Engineering  Framework  (AEF). 

Purpose 

The  AEF  is  being  developed  to  establish  a  structured  approach  with  tools  to  address 
program  affordability  challenges.  The  AEF  supports  the  USD(AT&L)  focus  on  restoring 
program  affordability  via  quantitative  analysis  of  the  products  in  the  portfolio  or  mission  area 
for  the  Technology  Development  Phase  and  trade  space  around  major  affordability  drivers 
in  the  Engineering  &  Manufacturing  Phase.  The  framework  provides  an  actionable  process 
for  program  managers  and  lead  engineers  to  assess  affordability  and  related  risks  and  to 
develop  courses  of  action.  While  useful  for  conducting  assessments,  the  framework  will  also 
identify  areas  where  affordability  engineering  and  analysis  need  to  be  inserted  in  program 
planning  and  execution.  Affordability  principles  of  efficiency  and  effectiveness  to  produce 
value  and  utility  need  to  be  applied  in  our  system  engineering  and  acquisition  management 
practices.  The  long-term  goal  is  to  provide  a  framework  for  establishing  an  affordability 
engineering  competency  among  systems  engineers  in  acquisition  programs  and  activities  to 
affect  timely  and  efficient  deliveries  of  capabilities  to  the  customer. 

Definitions 

The  following  terms  will  be  used  throughout  this  paper: 

•  Affordability  (At  the  program  level) — Acquire  the  user  need  within  the  budget, 
and  continue  to  fulfill  that  need  throughout  the  life  cycle  of  the  program. 

•  Efficient  (At  the  program  level) — Acquire  the  user  need  in  the  most 
economical  use  of  resources  (e.g.,  funding,  schedule,  staffing).  Provide 
greater  military  effectiveness  for  the  same  budget. 

•  Effective  (At  the  program  level) — Meet  or  exceed  the  operational  need  within 
budget  and  schedule. 

•  Program  Office  Estimate  (POE) — Provide  a  detailed  estimate  of  system 
acquisition  and  ownership  costs  normally  required  for  high-level  decisions. 
The  estimate  is  performed  early  in  the  program  and  serves  as  the  base  point 
for  all  subsequent  tracking  and  auditing  purposes. 

•  Technical  Baseline  (TB) — Provide  a  holistic  definition  of  the  system  and 
acquisition  program  accounting  for  all  aspects  that  relate  to  cost  and 
schedule.  The  TB  refers,  in  part,  to  the  characterization  of  the  physical  and 
functional  representation  of  intended  system  capabilities.  The  core  of  a  TB  is 
primarily  the  description  and  decomposition  of  hardware,  software,  and 
integration,  including  non-recurring  and  recurring  elements  that  make  up  the 
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system.  However,  much  more  is  needed  in  a  TB  to  support  life  cycle  cost 
analysis  in  affordability  engineering.  Technical  context  (such  as  system 
dependencies,  legacy  capability  migration  and  reuse,  technologies,  operating 
environment,  and  performance)  needs  to  be  understood.  A  description  of  the 
development  activities,  processes,  resources,  assets,  and  facilities  required 
to  engineer  the  system,  manage  the  acquisition,  perform  test  and  evaluation, 
and  ultimately  deploy  and  sustain  the  system  are  also  important  components 
of  the  TB.  Information  assurance  and  other  critical  engineering  constraints 
need  to  be  translated  into  development  activities  that  will  be  performed  and 
contribute  to  the  system  cost.  Similarly,  the  TB  must  fully  describe  production 
and  operations  and  must  support  phases  of  the  system. 

•  Tradeoff  Analysis — Evaluate  and  select  among  system  technical  functions, 
acquisition  strategy,  and/or  funding  alternatives  to  achieve  the  desired 
capabilities,  performance,  and  mission  effectiveness  within  cost  and  schedule 
objectives. 

AEF  Overview 

The  AEF  is  a  multi-step  framework  to  understand  a  program’s  affordability  risks  and 
challenges  and  to  provide  approaches  for  achieving  affordability,  efficiency,  and 
effectiveness  (AE&E)  in  an  acquisition  program.  Figure  1  illustrates  the  four  steps  in  the 
framework: 

•  Step  1 — Affordability  Risk  Assessment 

•  Step  2 — Affordability  Evaluation 

•  Step  3 — T radeoff  Analyses 

•  Step  4 — Assessment  and  Recommendations 


Figure  1.  AEF  Framework 

The  AEF  process  is  conducted  throughout  the  life  cycle  and  initiated  via  “trigger” 
points  that  occur  where  critical  program  management  activities  and  decisions  are  necessary. 
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These  trigger  points  include  periods  of  major  program  changes,  budget  preparation  and 
submittal,  and  existing  regulatory  and  statutory  requirements  for  affordability  certification.  A 
typical  program  profile  with  trigger  points  is  depicted  in  Figure  2. 
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Figure  2.  AEF  Program  Trigger  Points 

The  AEF  provides  a  significant  increase  in  the  number  of  affordability  assessments 
relative  to  current  requirements  as  there  would  be  four  or  more  before  Milestone  A  and 
seven  or  more  prior  to  Milestone  B.  The  increase  in  frequency  provides  two  major 
advantages:  (1)  stronger  coherency  from  assessment  to  assessment  and  (2)  assists  in 
institutionalizing  the  importance  of  affordability. 

Step  1 — Affordability  Risk  Assessment 

Step  1  is  a  qualitative  assessment  of  the  program  affordability  risk.  The  assessment 
is  accomplished  through  questionnaire  templates  that  refer  to  program  documentation  that 
would  comprise  the  TB  as  described  earlier.  The  assessment  includes  both  a  relative 
maturity  measure  among  the  various  TB  elements  and  a  maturity  comparison  with  the 
program  phase.  An  Excel-based  tool  referred  to  as  the  Affordability  Engineering  Risk 
Evaluation  (AERiE)  tool  is  being  developed  to  assist  in  the  assessment  task.  Figure  3 
illustrates  the  actions  taken  in  Step  1 . 
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Figure  3.  AEF  Step  1 — Affordability  Risk  Assessment 

The  first  step  in  using  the  AERiE  tool  is  assessing  the  program  environment  for 
managing  affordability.  There  are  five  program  conditions,  which  are  guiding  principles  of 
managing  for  affordability  and  foundational  to  each  trigger  evaluation.  The  program  team 
uses  the  following  five  categories  of  questions  to  assess  confidence  in  the  program 
management  conditions. 

1 .  Where  you  are  in  the  program  life  cycle  will  establish  the  maturity  of  the  TB. 
Given  the  current  phase  of  your  program,  is  the  program  TB  complete  and 
well  understood?  Is  it  updated  using  the  best  engineering  experience 
available  or  using  representative  analogy?  Is  it  documented  in  a  manner  that 
it  can  be  used  in  developing  sound  cost  estimates? 

2.  Again,  where  you  are  in  the  program  life  cycle  will  establish  the  fidelity  of  your 
cost  estimate.  Given  the  current  phase  of  your  program,  do  you  believe  the 
cost  estimate  is  sound?  Is  it  well-documented  and  based  on  a  complete  and 
well-indentured  work  breakdown  structure?  Was  it  developed  using 
reasonable  data  and  analogies?  Does  it  have  engineering-based  inputs  that 
include  ranges  that  reflect  program  risk?  Have  appropriate  costing  methods 
been  used?  Is  the  program  implementation  based  upon  costs  that  are 
reflective  of  program  risks? 

3.  Was  the  program  schedule  developed  using  the  same  TB  as  the  program 
cost  estimate? 

4.  Does  the  program  have  a  disciplined  approach  to  risk  and  requirements 
management  supported  by  system  engineering  and  associated  cost 
analysis? 

5.  Are  the  program  risk,  cost,  schedule,  and  requirements  management 
integrated  so  that  the  cost  estimates  always  reflect  the  latest  risk  mitigations, 
schedule  changes,  and  requirements  baseline  decisions? 
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After  completing  program  condition  “goodness”  checks,  the  program  team  is  directed 
to  proceed  to  the  AERiE  instruction  page  and  begin  the  AERiE  assessment.  The  program 
team  should  consider  the  five  program  conditions  when  rating  the  selected  AERiE 
assessment  templates.  Each  trigger  has  a  unique  affordability  assessment  template  (see 
Figure  4). 


Figure  4.  Typical  AERiE  Template 


Each  template  contains  unique  assessment  questions  that  address  affordability  risk 
indicators  contained  in  the  program’s  technical  baseline. 


For  each  question,  the  user  selects  a  risk  level  (i.e.,  high,  medium,  low,  unknown, 
and  not  applicable);  unique  risk-level  definitions  are  provided  for  each  trigger  question.  Upon 
selecting  a  risk  level,  the  tool  provides  question-specific  recommendations  for  possible 
corrective  or  mitigation  actions.  Unique  recommendations  are  provided  for  each  question’s 
risk  level.  An  assessment  tally  is  provided  upon  completion  of  the  assessment. 


The  assessment  result  provides  the  program  team  with  evidence  of  risks,  indicating 
the  state  of  the  program’s  affordability  position.  In  addition,  the  tool  warns  of  “show 
stoppers”  that  prevent  the  team  from  understanding  the  program  affordability  position  as  a 
result  of  the  assessment.  Corrective  or  mitigating  changes  are  recommended,  unless  all  the 
template  questions  are  assessed  as  low  risk.  Once  the  changes  are  complete,  the  program 
team  will  validate  the  TB  and  the  POE  in  Step  2  in  the  process  of  completing  a  quantitative 
affordability  evaluation. 
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Step  2 — Affordability  Evaluation 

Step  2  determines  the  program’s  affordability  in  a  quantitative  manner  (see  Figure  5). 
This  is  accomplished  by  making  changes  as  necessary  to  validate  the  TB  and  the  POE  for 
completeness  and  accuracy.  Once  valid,  the  POE  and  the  associated  schedule  are 
compared  to  the  existing  program  budget. 


TBF/CE  -  Technical  Baseline  Framework/Cost 
EngineeringGuide 
TB  -  Technical  Baseline 
POE  -  Program  Office  Estimate 
C/A/R  -  Complete/ Accuracy/Realism 


BUDGET  AND 
SCHEDULE 
ARE 

SUFFICIENT 


Figure  5.  AEF  Step  2 — Affordability  Evaluation 

The  state  of  the  program  foundation  is  examined  before  beginning  the  TB  validation 
process.  This  ensures  that  the  TB  and  POE  emerging  from  Step  2  will  be  reliable.  The 
components  of  this  evaluation  include  the  following: 


•  Continuous  Cost  Engineering — The  program  should  have  a  disciplined 
approach  to  risk  and  requirements  management,  which  includes  tight 
integration  of  risk,  cost,  schedule,  and  requirements  management  and 
coordination  with  users  and  other  active  stakeholders.  This  will  result  in 
reliable  tradeoffs  and  program  cost  estimates  that  reflect  the  latest  risk 
mitigations,  schedule  changes,  strategy  updates,  and  requirements  baseline 
decisions.  As  a  corrective  action,  the  program  should  improve  affected 
processes  (e.g.,  requirements  definition,  system  design,  program  planning),  if 
needed,  while  continuing  with  AEF  Steps  2-4. 

•  Soundness  of  Program  Cornerstones — The  program  should  have  acquisition 
artifacts  and  engineering  and  management  products  that  are  consistent  with 
its  maturity  in  the  acquisition  life  cycle  and  the  requirements  of  DoD 
Instruction  5000.02.  AEF  users  informally  consider  the  state  of  their  program 
relative  to  a  description  of  the  desired  elements  in  an  acquisition  and  look  for 
high-level  issues.  Corrective  actions,  if  needed,  include  (1)  improving 
outreach  to  users  and  acquisition  stakeholders  by  identifying  their  latest 
needs  or  changed  acquisition  context  and  (2)  completing  or  updating  artifacts 
and  incorporating  changes  in  the  TB/POE/schedule. 
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•  Resolution  of  Harmful  Trends  Revealed  by  Step  1  Affordability  Risk 
Assessment — An  affordability  risk  trend  is  revealed  from  risks  identified 
across  different  Step  1  triggers  because  of  the  coherency  from  assessment  to 
assessment.  The  trend  has  an  underlying  cause  or  connection  among  the 
risks  that  might  hinder  the  program  in  delivering  affordable,  timely,  and 
effective  capability  to  the  warfighter.  The  revelation  of  a  trend  is 
accomplished  by  the  AEF  user  compiling  all  show  stopper  and  potential  show 
stopper  risks  and  studying  these  risks  to  reveal  common  themes/causes.  The 
AEF  will  provide  exemplars  by  program  phase  or  milestone,  relating  them  to 
alarms/recommendations  from  different  triggers.  Each  trend  will  reveal,  by  the 
nature  of  the  trend,  the  corrective  action  needed  and  the  frequency  with 
which  it  should  be  applied. 

The  TB  is  evaluated  for  completeness  by  using  a  Technical  Baseline  Framework  and 
Cost  Engineering  (TBF&CE)  Guide  ( TBF&CE  Guide,  2011)  in  development.  Evaluation  of 
the  TB  involves  an  element-by-element  comparison,  illustrated  in  Figure  6,  of  the  current  TB 
to  the  TB  checklist  and  the  TBF&CE  Guide,  following  the  TB  validation  process. 
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Figure  6.  TB  Element-by-Element  Comparison  Example 

The  TB  is  valid  if  it  is  complete  (i.e.,  contains  all  the  elements  of  the  TB  framework), 
characterizes  all  the  cost  elements  (i.e.,  contains  or  points  to  the  data  required  to  cost  every 
element),  and  has  the  appropriate  level  of  definition  and  fidelity  for  the  point  in  the  life  cycle 
(i.e.,  reflects  the  maturity  of  the  system  design,  sustainment  approach,  and  acquisition 
strategy  and  portrays  that  maturity  realistically). 

If  the  program  office  has  no  TB  or  their  TB  is  invalid,  the  AEF  process  directs 
corrective  action  to  update  or  build  a  program  TB.  The  TBF&CE  Guide  contains  a  “build  your 
own  workspace”  template  that  guides  the  program  through  a  specific  correction  to  the  TB  or, 
if  needed,  the  construction  of  a  complete  TB. 

Once  the  TB  has  been  validated,  or  corrected  and  validated,  the  program  team  can 
proceed  with  the  evaluation  and,  if  needed,  the  iteration  of  the  POE.  If  the  TB  requires 
updating,  the  POE  must  be  revised  to  be  consistent  with  the  TB  changes  prior  to  the  team’s 
validating  it. 

POE — A  detailed  estimate  of  system  acquisition  and  ownership  costs 
normally  required  for  high-level  decisions.  The  initial  estimate  is  performed 
early  in  the  program.  It  serves  as  the  base  point  for  all  subsequent  tracking 
and  auditing  purposes. 

Then  the  POE  is  valid  if  it  is  complete  (i.e.,  estimates  costs  for  all  elements  of  the 
TB),  realistic  (i.e.,  identifies  costs  considering  available  data),  and  reasonable  (i.e.,  accounts 
for  TB  assumptions  and  associated  risks). 
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The  POE  is  evaluated  using  the  POE  Validation  Guidance  (PVG)2  and  products 
derived  from  the  guide  (e.g.,  Program  Office  Estimate  Validation  Process  and  Program 
Office  Estimate  Validation  Checklist). 

If  the  POE  is  not  valid,  the  AEF  process  directs  corrective  action  to  update  the  POE 
and  revalidate,  if  required.  This  activity  is  iterative  and  intended  to  reveal  information  that 
may  require  additional  updates  to  the  TB. 

Once  the  POE  is  validated,  the  AEF  process  directs  a  comparison  with  the  program 
budget.  With  a  validated  TB  and  POE,  the  POE  and  estimated  schedule  are  compared  to 
the  program  budget  and  program  schedule. 

•  Program  is  affordable.  If  the  budget  or  program  schedule  is  sufficient  relative 
to  the  POE  and  estimated  schedule  (i.e.,  the  program  is  affordable),  the 
program  team  can  exit  the  process  or  continue  to  Step  3  (recommended)  for 
discovering  potential  efficiency  and  effectiveness  improvements. 

•  Program  is  unaffordable.  If  the  budget  and/or  program  schedule  is  insufficient 
relative  to  the  POE  and  estimated  schedule,  the  program  team  will  proceed  to 
Step  3  to  identify  corrective  action  alternatives  and  potential  efficiency  and 
effectiveness  improvements. 

The  validation  of  the  TB  and  POE  will  reveal  technical,  performance,  schedule, 
acquisition,  and/or  logistics  drivers  of  the  program.  These  elements  have  a  strong  influence 
on  the  feasibility  and  affordability  of  the  program.  These  drivers  may  directly  or  indirectly 
drive  the  program’s  cost  and  schedule  or  impose  risk  that  should  be  mitigated.  Leaving  Step 
2,  the  program  team  will  have  identified  the  program  effectiveness  and  cost  drivers  as  well 
as  a  quantitative  affordability  evaluation.  These  program  drivers  will  become  the  subjects  of 
tradeoff  analyses  conducted  in  Step  3. 


2  PVG  is  a  set  of  authoritative  cost-estimating  and  evaluation  documents  that  have  been  placed  in  the  MITRE 
SEPO  Cost  Estimating  Toolkit. 
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Step  3 — Tradeoff  Analyses 


Initiate  Trade  Study  Process  for  Next  Driver  (Repeat  Process  Until  All  Drivers  are  Addressed) 
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Figure  7.  AEF  Step  3 — Tradeoff  Analyses 

Step  3  is  designed  to  develop  and  conduct  structured  tradeoff  analyses  and  walks 
the  program  team  through  a  deliberate  process.  Much  of  this  step  is  formative  to  the  tradeoff 
analysis  process,  identifying,  structuring,  evaluating,  and  determining  candidate  trade 
studies  that  should  move  to  Step  4  for  final  analysis. 

Tradeoff  Analysis — The  process  of  evaluating  and  selecting  among  system 
technical  features,  acquisition  strategy,  and/or  funding  alternatives  to  achieve 
the  desired  capabilities,  performance,  and  mission  effectiveness  within  cost 
and  schedule  objectives. 

In  Step  3,  the  program  team  reviews  the  trade  study  cost  and  effectiveness  drivers 
that  were  identified  in  Step  2.  This  ensures  that  the  team  understands  the  affordability 
issues/challenges  and/or  AEE  opportunities  with  a  program  life  cycle  perspective. 

For  each  of  the  drivers,  the  user  defines  the  integrated  (cost,  schedule,  performance) 
trade  space3  that  needs  to  be  examined  and  the  candidate  trades  that  may  exist  in  that  trade 
space.  If  the  budget  or  program  schedule  is  sufficient  relative  to  the  POE  and  estimated 
schedule  (i.e.,  the  program  is  affordable),  the  user  will  focus  on  trade  opportunities  that  can 
be  analyzed  to  achieve  cost  savings/avoidance  and/or  to  improve  the  effectiveness  of  the 
system  required.  If  the  budget  or  program  schedule  is  insufficient  relative  to  the  POE  and 
estimated  schedule  (i.e.,  the  program  is  unaffordable),  the  user  will  focus  on  cost/schedule 
reduction  trades  that  will  allow  the  program  budget  to  be  sufficient  (i.e.,  deliver  the  warfighter 
capability  within  the  program  budget).  The  generic  trade  study  process  is  shown  in  Figure  8. 


3  Trade  Space — The  multivariable  set  of  fiscal,  temporal,  legal,  political,  operational,  sustainment,  program  and 
system  parameters,  attributes,  and  performance  characteristics  required  to  satisfy  user  needs  that  are  used  by 
decision-makers  to  make  informed  and  structured  program  or  portfolio  decisions. 
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Generic  Trade  Study  Process* 


*  Based  on  Decision  Analysis  Process  work  by  Duquette  &  Bloom  2008 


Figure  8.  Generic  Trade  Study  Process 

The  trade  study  process  is  an  overarching  process  for  Step  3.  Detailed  actions  are 
specified  for  each  step  to  guide  the  program  team  through  a  rigorous  evaluation  of  the  trade 
alternatives.  The  next  series  of  actions  in  Step  3  is  the  first  stage  of  the  trade  study  analysis. 

The  user  first  selects  the  Trade  Study  Analysis  Paradigm  and  Checklist.4  Four 
paradigms  and  checklists  will  be  available  for  the  program  team: 

•  Features/Functions/Performance  (F/F/P) 

•  Operations  and  Support 

•  Acquisition  Strategy 

•  Life  Cycle  Funding 

The  user  then  identifies  the  trade  options  and  selection  criteria  through  a  set  of 
defined  methods.  For  example,  one  trade  in  F//P  might  be  a  non-development  item  (NDI) 
versus  development. 

The  next  action  (Step  9  in  Figure  8)  specifies  a  set  of  alternative  solutions  that  will 
satisfy  the  challenges/issues  and/or  opportunities  posed  by  the  driver. 

The  tradeoff  alternative  solutions  are  then  evaluated  for  feasibility  and  compatibility. 

A  feasibility  determination  is  accomplished  by  using  feasibility  verification  elements. 
Figure  9  is  an  example  of  the  feasibility  verification  elements  for  an  acquisition  strategy.  For 
example,  if  the  production  approach  (Number  13)  is  far  left  or  right  on  the  scale,  the 
acquisition  strategy  trade  is  significantly  constrained. 


4  The  Trade  Study  Analysis  Paradigm  and  Checklist  provide  the  context  and  actions  necessary  to  carry  out  a 
trade  study  for  the  four  paradigms. 
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|  22.  TECHNOLOGY  MATURITY 

SLEP/Mod 

New  Technology 

23.  TEST  APPROACH 

Mature  Technology 

Integrated  Testing 

Serial  Testing 

Figure  9.  Tradeoff  Alternative  Feasibility  Verification  Example 


If  the  trade  is  not  feasible,  the  program  team  selects  an  alternate  trade.  If  the  trade 
study  is  feasible,  a  compatibility  check  against  the  validated  TB  is  performed.  The  program 
team  identifies  adjustments  to  the  TB,  enabling  the  specific  trade  and  its  alternative 
solutions.  For  example,  if  the  trade  is  an  F/F/P  trade,  adjustments  may  be  required  to  the 
acquisition  strategy,  sustainment  strategy,  funding  profiles,  and  so  forth  to  implement  the 
alternate  solutions.  All  collateral  impacts  and  adjustments  to  the  TB  relating  to  the  trade  are 
identified. 


Compatibility  is  now  evaluated  by  subject-matter  experts.  The  process  determines 
tradeoffs  that  need  to  be  bundled  due  to  coupling  among  individual  trades.  The  changes  to 
the  TB  are  also  determined.  Each  tradeoff  bundle  will  consist  of  a  combination  of  compatible 
feasible  tradeoffs  (i.e.,  the  applicable  set  of  the  four  Trade  Study  Analysis  Paradigms  and 
their  collateral  TB  adjustments). 

Tradeoff  bundle  elements  are  compatible  if  the  combination  of  feasible  trades  and 
collateral  TB  adjustments  provide  a  workable  alternative  to  the  existing  condition.  If  the 
tradeoff  bundle  is  compatible  and  can  be  implemented  at  the  program  level,  the  next  action 
in  Step  3  is  to  document  the  analysis  and  the  recommended  feasible  tradeoff  bundle.  If  the 
tradeoff  bundle  is  better  suited  to  a  portfolio  implementation,  it  is  deferred  to  the  portfolio 
level  for  consideration.  If  the  tradeoff  bundle  is  not  compatible,  it  is  revised,  if  possible,  and 
reevaluated  for  compatibility. 

The  user  repeats  the  Step  3  process  to  generate  feasible  and  compatible  tradeoff 
bundles  for  each  of  the  cost  and  effectiveness  drivers  that  were  identified  in  Step  2. 

Step  4 — Assessment  and  Recommendations 

The  objective  of  Step  4  (Figure  10)  is  to  select  efficiently  and  effectively  the  tradeoff 
bundles  that  deliver  the  capabilities  that  the  end  user  needs  within  the  established  budget 
and  time  line.  These  recommendations  are  based  upon  benefit,  risk,  cost,  and  schedule 
impacts. 
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Figure  10.  AEF  Step  4 — Assessment  and  Recommendations 

If  the  program  was  determined  unaffordable  in  Step  2,  the  program  team  assesses 
feasibility/effectiveness  of  the  various  trade  bundles  established  in  Step  3  to  define  an 
affordable  program.  If  the  program  was  determined  affordable  in  Step  2,  the  program  team 
will  evaluate  the  tradeoff  bundles  to  improve  the  affordability  position  of  the  program  through 
improvements  in  efficiency  and  effectiveness. 

Recommendations  are  made  by  considering  the  tradeoff  bundles’  or  set  of  bundle 
alternatives’  ability  to  meet  affordability  goals,  efficiency  in  meeting  these  goals,  and  the 
effectiveness  in  delivering  needed  mission  capabilities. 

In  Step  4,  conducting  the  analysis  of  each  tradeoff  bundle  requires  the  following 
costing  and  evaluation  activities: 

•  evaluate  the  risk  of  the  trade  bundle  by  applying  the  program  risk 
management  process,5 

•  determine  the  costs  associated  with  the  risks  identified  in  the  bundle, 

•  determine  the  cost  and  risk  of  implementing  the  tradeoff  bundle,6 

•  determine  net  cost  savings  if  the  bundle  was  implemented,  and 

•  determine  the  benefit  of  the  bundle. 


5  Use  the  risk  management  process  like  the  following:  http://www.mitre.org/work/sepo/toolkits/risk/ 

6  Bundled  risk  cost  can  be  evaluated  with  Paul  Garvey’s  (2012)  Measuring  Economic  Returns  of  Risk-benefit 
Tradeoffs  in  System  Acquisitions — An  Affordability,  Effectiveness,  and  Efficiency  Perspective. 
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There  are  a  variety  of  assessment  tools  being  evaluated  that  can  be  adapted  to 
evaluate  the  cost,  risk,  and  benefit  data  that  has  been  developed  in  this  step.  Available  tools 
include  the  following: 

•  Portfolio  Analysis  Machine  (PALMA™), 

•  Desk  Top  Return  on  Investment, 

•  Kepner-Tregoe  Method,  and 

•  Investment  Portfolio  Analysis  Model  (IPAM). 

The  relationships/dependencies  among  AE&E  are  illustrated  on  a  cost-benefit 
graphic  in  Figure  1 1 .  The  Figure  1 1  efficient  frontier  curve  describes  the  most  efficient  state 
possible  for  a  given  benefit/cost  combination.  The  trade  space  is  the  area  of  the  graphic 
bounded  by  the  efficient  frontier  and  the  effectiveness  and  affordability  vectors.  The 
objective  is  to  move  in  the  direction  of  increased  efficiency,  which  can  be  quantitatively 
measured  by  the  assessment  tools  identified  earlier.  Devising  solutions  to  address 
affordability  challenges  requires  understanding  what  drives  each  dimension  and  how 
developing  alternatives  in  the  different  tradeoff  paradigms  can  move  a  program  along  a 
specific  vector.  The  AEF  is  being  designed  to  enable  exploration  of  this  AE&E  trade  space 
while  providing  how-to  guidance  for  identifying  and  addressing  affordability. 


Cost 


Figure  11.  AE&E  Relationships 

The  bundled  tradeoff  alternatives  are  evaluated  against  the  program  cost  position  for 
improved  efficiency  and  affordability.  Recommendations  are  provided  to  decision-makers  for 
determination.  If  the  decision-makers  accept  the  recommended  set  of  alternative  tradeoff 
bundles,  the  program  team  designs  the  implementation. 

Status  and  Implementation  of  the  AEF 

The  AEF  development  is  planned  to  be  completed  by  September  2012.  The  step¬ 
wise  framework  permits  incremental  evaluation  via  “piloting”  within  existing  acquisition 
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programs  prior  to  completion.  The  piloting  activity  is  planned  for  June  through  September 
2012  within  representative  programs  across  the  Navy,  Army,  and  Air  Force.  To  facilitate  the 
piloting,  a  quick-start  guide  will  be  developed. 

The  actual  implementation  will  vary  from  program  to  program  but  will  have  the 
following  common  tenets: 

•  a  single  technical  baseline  definition  for  cost,  schedule  and  performance 
planning,  modeling,  executing,  and  reporting, 

•  incorporation  of  cost  and  schedule  into  the  traditional  engineering  trade 
space, 

•  leverage  of  the  integrated  trade  space  to  develop  bundled  tradeoff 
alternatives  for  program  decision-makers, 

•  actionable  framework  with  appropriately  detailed  tools,  and 

•  execution  via  integrated  system  engineering  and  cost  analyst  teams. 

The  program  systems  engineering  and  financial  management  processes  will  require 
modification  for  implementation  of  these  tenets.  As  with  most  changes,  successful 
implementation  will  require  priority  from  program  leadership.  The  AEF  crosses  multiple 
disciplines  and  should  be  led  by  the  program  manager  (PM).  The  outcome  of  a  successful 
AEF  implementation  is  an  execution  that  will  be  measurably  more  efficient  in  the  dimensions 
of  affordability  (cost)  and  effectiveness  (benefit). 

Summary 

The  AEF  is  designed  to  provide  a  rigorous  approach  for  proactively  achieving 
program  affordability.  The  AEF  supports  the  USD(AT&L)  mandate  to  restore  acquisition 
program  affordability  and  control  cost  growth.  It  does  so  by  a  multi-step  process  that 
qualitatively  and  quantitatively  measures  program  affordability  risk,  developing  a  set  of 
targeted  tradeoffs  that  are  bundled,  evaluated  for  compatibility,  and  then  recommended  for 
implementation.  The  AEF  is  designed  to  be  actionable  with  tools  and  templates  to  guide 
program  teams  during  execution.  The  AEF  is  conducted  in  a  manner  that  increases  the 
frequency  of  affordability  assessment  to  improve  assessment  quality,  with  integrated  multi¬ 
disciplinary  program  teams  to  institutionalize  the  management  of  integrated  cost,  schedule, 
and  performance  trade  space.  The  primary  objective  is  to  increase  the  probability  of 
program  success  in  a  challenging  budget  environment  through  increased  execution 
efficiency  throughout  the  life  cycle  and  provide  program  managers  with  data-driven  rationale 
for  program  change  recommendations. 
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Overview 


■  Program  Affordability  is  of  paramount  importance  in 
the  current  fiscal  environment. 

■  MITRE’s  Affordability  Engineering  Framework  (AEF) 
Project  aims  to  develop  a  systems  engineering 
process  to  address  Program  Affordability. 

-  Goal:  Shape  program  to  achieve  BBP  “should  cost”  and 
address  affordability  challenges. 

-  Maturity:  ~60%  overall;  development  scheduled  to 
complete  Sep  ’12. 

-  Currently  identifying  pilot  programs  to  shape  and  validate  AEF 
tools  and  techniques. 


The  AEF  can  benefit  the  PMO  by  providing  an  actionable  process  to 

proactively  manage  program  affordability. 
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AEF  Objectives 

Provide  a  standardized,  actionable,  systems  engineering  approach  to 

make  programs  more  affordable  in  execution. 

■  Improve  government  program  technical  and  budget  planning. 

Develop  a  government  technical  reference  design. 

Requirements  feasibility,  cost/performance  modeling,  RFP  preparation  and  proposal  evaluation. 

■  Reduce  “uncertainty”  in  cost-estimating  to  mitigate  affordability  risk. 

Build  a  comprehensive  program  baseline  based  on  the  reference  design  - 
“Acquisition  Systems  Engineering  Baseline”  -  similar  to  CARD. 

Frequently  iterate  the  ASE  baseline  as  a  “living  document”  to  tightly  couple 
PMO  cost  analysis  and  technical  activities. 

■  Develop  integrated  program  trade-space  for  cost,  schedule  and 

performance  to  construct  COAs  to  address  affordability. 

Provide  data  driven  analytical  products  for  more  accurate  and  defendable  PMO 
cost  positions  and  trade  offs. 

Emphasize  integrated  systems  engineering  and  cost  estimating  activities. 


Institutionalize  “cost  consciousness”  in  PMO  decision-making. 
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The  Affordability  Engineering  Framework 

■  Multi-step  process  conducted  iteratively  throughout  the 
program  lifecycle. 


STEP  1  -  Affordability  Risk  Assessment 


» Conduct 
Affordability  Risk 


n 


Assessment 


STEP  4  -  Assessment  &  Red 


© 


Nation 


4.  Evaluate,  Select 
and  Implement 


STEP  2  -  Affordability  Evaluation 

2.  Conduct 
q-  Quantitative 
Affordability 


Evaluation 


I  I 

(3) 


d> 


I  --- 1  COA  s* 

□ 

Analyses 

i 

P_ 

STEP  3  -  Tradeoff  Analysis 

!4>-<™r> - 


s 

: 

. ak'%"  1 

S.  Conduct 
Tradeoff 
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AEF  in  the  Program  Lifecycle 


One  Time 


One  Time  with  Potential  Updates  or  Multiple  Times 


Technology 

Development 


Engineering  &  Manufacturing 
Development 


AEF  initiated  by  “triggers”  that  map  to  critical  acquisition 
engineering/management  activities  and  decision  points: 

-  Program  changes 

-  POM  cycles 

-  Regulatory  and 
statutory 
requirements 

More  frequent 
assessments 

(beyond  the  current 
regulatory  and  statutory 
requirements) 

designed  to  provide  better  affordability  “situational  awareness” 
and  coherency  between  measurements. 


Integrated  System 
Design 


System 

Eipmonsfration 

o 


~ioc1(g(p 

FOC 

Production  & 

Operations  & 

Deployment 

Support 

1  LRIP 

Su  s'-a  inirtprit  & 

■oifiE 

Disposal 

0  3-KP 

Decision  Review 

_ A 

FOC 

Recurring  (Unpredictable) 


Increased  affordability  “situational  awareness”  for  improved  program 

decision-making. 
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AEF  Step  1  -  Affordability  Risk  Assessment 


Oi*  Ti-n»  wilh  Prfp-iliit  UpdjIjiCT  Itfuhiplp  Tmn 


. . .  ■■; 


ft* i u rrin;i;UrcTrf,:1jaj  I 


□  ''Quick  look"  using  an  excel-based 
tool  -  "AERiE"  . 

□  Templates  designed  for  each 
"trigger"  point  derived  from 
lessons-learned  and  SME 
recommendations. 

□  Interview  and  evaluate  program 
information/documentation... 

>  "Maturity  Assessment": 

Content  detail? 

>  "Confidence  assessment": 
Process  and  content  quality? 


Affordability 

Engineering 

Risk 

Evaluation 
AERiE  Tool 


□  Output(s): 

>  Affordability  Risk  Assessment. 

>  Partial  Acquisition  Systems 
Engineering  (ASE)  Baseline 
(analogous  to  DoD  CARD  data) 
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AEF  Step  2  -  Affordability  Evaluation 


□  Conduct  quantitative  evaluation 
of  the  program  affordability 

□  Assemble  a  comprehensive 
"Acquisition  Systems  Engineering 
(ASE)  Baseline"  ..similar  to  CARD. 

□  Emphasis  on: 

>  Multi-discipline 
Teams 

>  Detailed  Core 
Technical  design 

>  Risk  mitigation. 

>  Modeling. 

>  Program 
Interdependency. 

>  Acq  Strategy. 


MITRE 

The  Acquisition  System  Engineering 

Baseline  Framework  and  Cost 

Engineering  Guide 

Coordination  Draft 

_ ^ 

D.  a.  cwine 

F  ■  MtoRuwo 

DRAFT 

ASE 

Purpose: 

Program  Origins 

Baseline 

Who  needs  this 

Motivation, 

k^UWV  1  1  1  IV 

acquisition? 

Predecessor  system(s) 

User  requirements: 

Capabilities,  Functions, 
Performance,  Schedule 


\ 


System  Overview 

System/functional  description 
Ref.  Architecture, 
Capabilities,  KPPs, 


Collateral  Efforts 


External  relationships: 

Architecture,  dependencies, 
interfaces 


Core  Technical 
Baseline 

System  decomposition: 
subsystems,  HW  &  SW 
components,  COTS 


System  Engineering 


Specialty  Engineering 


Management 


Enterprise  Considerations: 

Legacy  systems/components, 
Engineering  Baseline,  Technical 
Standards,  Interoperability 


Integration  (multiple  levels) 


System  Test,  (OT&E) 


Design,  Development, 
Engineering,  Efforts 


Operational/Maint.  Concept 


Qual  Test,  DT&E 


Logistics 

System 

Deployment 

Platform 

Integration 

Technology  Strategy:  1^ 

maturity,  insertion,  refresh 
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Operations 


Acquisition  Approach:  Strategy,  Schedule,  Contracting 


□  Evaluate  and  iterate  the  Program  Office 
Estimate  (POE). 

□  Compare  the  POE  to  the  existing  program 
budget. 

□  Outputs: 

>  Quantitative  affordability  position. 

>  Completed  ASE  Baseline  and  POE. 

>  Integrated  C/S/P  trade  space. 

>  Cost  drivers  and  uncertainty. 


MITRE 
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AEF  Step  3  -  Tradeoff  Analysis 


□  Leverage  integrated  C/S/P  trade  space  to 
develop  and  analyze  trade  offs. 

□  Structured  trade  study  analysis  process: 
constraints  assumptions,  evaluation 
criteria,  weighting. 


□  Tradeoff  types  determined  by 
primary  driver: 

>  Features,  functions, 
performance 

>  Operations  and  support 

>  Acquisition  strategy 

>  Life  Cycle  Funding 

□  Each  tradeoff  is  measured  for: 

>  Effectiveness 

>  Cost 

>  Schedule 

>  Risk  scoring 

>  Sensitivity  analysis 

>  Dependency  (i.e.,  change 
compatibility) 

□  Output: 

>  Tradeoff  Analysis  Summary 
Table 


\  ^ 

Critical  Criteria 

|  Non-Critical  Criteria 

Relative  Effectiveness 

Score 

"Raw  Decision  Criteria" 

Comments 

Criteria 

Alternatives  1  \ 

Speed 

Weight  =  .22 

Support  Vendors 

W<*paJo7 

rbf 

a!  o 

m 

[  ?  s 

m 

[  §  5 

Tech  ifl^rer  Rate 
Weu^=  -04 

I  . 

jre 

Lb 

Total 

Ownership 
Cost*  in  FY14 
Million  $ 

Schedule 
in  Months 

Risk 

(P) 

Ba£Mne  Case 

ScoM^VeigtnMtore 

ActSWalue 

0.60  0.13 

1.00  |  0.07 

4.00  0.60 

0.40  0.12 

0.80  |  0.16 

2.00  0.14 

4.90  0.20 

4.00  0.36 

1.73 

1465 

24 

69.00 

Baseline  System  for 
Effectiveness 
Comparison 

23  mph 

1  vendor 

TRL9 

Ao  =  72% 

195  mi 

9  level 

20.7  mo 

One  Sys 

Altiyjlive  1  fQ 
Sco^^Veigh1^2core 

Actfnalue 

2.40  0.53 

3.00  0.21 

1.00  0.15 

2.90  0.49 

3.10  0.62 

2.00  0.14 

4.90  0.20 

1.00  0.09 

2.43 

1155 

36 

58.93 

39  mph 

3  vendors 

TRL  6 

Aq  =  84.5% 

252.5  mi 

9  Level 

20.7  mo 

Dev  Comp 

A'tJnSi ve2  fll 

St^^^Veigh^^ypore 
Actual  Value™^* 

4.50  0.99 

3.00  |  0.21 

2.00  0.30 

3.80  0.65 

3.20  |  0.64 

3.00  0.21 

3.10  0.12 

5.00  0.45 

3.57 

1071 

21 

47.14 

60  mph 

3  vendors 

TRL  7 

A0  =  89.5% 

255  mi 

7  Level 

12.3  mo 

NSA  Prod 

Alternative 

Score  /  Weighted  Score 

Actual  Value 

4.50  0.99 

1.00  |  0.07 

2.00  0.30  3.20  0.64 

1.00  0.07 

2.40  0.10 

3.00  0.27 

2.44 

1605 

18 

72.02 

60  mph 

1  vendor 

TRL  7  255  mi 

1  Officer 

10.2  mo 

Acer  Pend 

Tradeoff  Alternatives  Analysis  Summary  Table 
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AEF  Step  4  -  COA  Selection  and  Implementation 


□  Evaluate  the  candidate  COAs 
for:  affordability  targets, 
mission  effectiveness,  and 
efficiency. 

□  Benefit  scores  are  normalized 

values  from  decision  factors. 
>  Acceptable  score 
determined  from 
effectiveness  measures. 

□  Cost  score  from  analyses  . 

□  Both  benefit  and  cost  scores 
incorporate  uncertainty 
ranges. 


□  Select  a  COA  and  develop  the 
implementation  plan. 

□  Output: 

>  Decision  to  execute  Course  Of 
Action  to  achieve  affordability 
objective. 

>  Initial  implementation  plan. 
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Takeaways  to  Improve  Program  Affordability 


Institute  a  data-driven  SE  process  to  measure  program  affordability  and 

manage  to  “ should  cost”. 

■  Develop  and  maintain  a  government  reference  technical  design  to 
strengthen  government  program  technical  team. 

Use  for  requirements  realism,  cost  estimating/modeling,  proposal  risk 
evaluation. 

■  Use  a  comprehensive  “costable”  program  baseline  (e.g.  CARD  or 
ASE)  and  iterate  it  frequently  to  maintain  an  accurate  cost  estimate. 

Align  cost  models,  technical  configurations  and  performance  models. 

■  Develop  and  leverage  integrated  C/S/P  program  trade  space  for 
COAs  to  respond  to  budget  challenges. 

■  Conduct  the  process  with  integrated  Systems  Engineering  and  Cost 
Analysis  teams. 


Adopt  a  rigorous  Affordability  Engineering  approach  to  “exercise 

more  disciplined  use  of  defense  dollars”. 
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Back-Up 
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AEF  Step  1 

AERiE  Tool 


Affordability  Risk  Assessment 

Lifecycle 


Acquisition 

Systems 

Engineering 

Baseline 


Chart  Legend 

Can't  Reasonably  be  Defined 

TT 

Assumptions 

rn 

Early  Approximation 

Preliminary  Definition 

_ 

Improved  Definition 

Stable  Definition 

_ 

X 


X 


X 


31 


10 


32 


Risk  Management 


annlieri  in  production  cost 


Is  the  risk  management 
system  described  in  the 
acquisition  strategy  managed 
independently  by  the 
Government? 


Are  the  competition  strategy, 
contracting  approaches. _ 


Risk  Levels  for  this  Test: 

H  =  The  risk  management  system  is  performed  by  a  prime  contractor  or  another  organization  with  a  pecuniary  stake  in  the  actions  of  the  risk 
management  team. 

M  =  There  is  some  form  of  jointly  run  risk  management  between  the  Government  and  a  contractor. 

L  =  The  Government  or  an  objective  party  runs  risk  management. 

U  =  It  is  not  known  who  runs  risk  management  (or  if  there  is  risk  management). 

NA  =  There  is  no  need  for  risk  management,  or  risk  can  be  managed  effectively  by  any  party. 
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The  system  description  and  characteristics,  program  definition,  and 
acquisition  approach  that  account  for  all  aspects  of  a  program  relevant  to  cost 
and  schedule 

Developed  by  a  cross-functional  program  team 

Used  to  perform  engineering  trade-offs  and  estimates  of  all  types  in  support  of 
acquisition  decisions 


ASE 

Baseline 


System  Overview: 

System/functional  description 
Ref.  Architecture, 
Capabilities,  KPPs, 


Purpose: 

Who  needs  this 
acquisition? 

~ r 


Program  Origins: 

Motivation, 

Predecessor  system(s) 


External  relationships: 

Architecture,  dependencies, 
interfaces 


Enterprise  Considerations: 

Legacy  systems/components, 
Engineering  Baseline,  Technical 
Standards,  Interoperability 


Operational/Maint.  Concept 


Technology  Strategy: 

maturity,  insertion,  refresh 


Core  Technical 
Baseline 


System  decomposition: 
subsystems,  HW  &  SW 
components,  COTS 


Integration  (multiple  levels) 


Design,  Development, 
Engineering,  Efforts 


Qual  Test,  DT&E 


TV 


A 


User  requirements 

Capabilities,  Functions 
Performance,  Schedule 


Collateral  Efforts 


n\ 


System  Engineering 


Specialty  Engineering 


Management 


System  Test.  (OT&E) 


Logistics 


System 

Platform 

Deployment 

Integration 

Operations 


Acquisition  Approach:  Strategy.  Schedule.  Contracting 


MITRE 


Being  matured  via 
the  Affordability 
Engineering 
Framework  (AEF) 
Capstone  Project 
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Relative  Scope  of  the  ASE  Baseline 


Program  Baseline:  entirety  of  program  strategies,  concepts,  goals 


ASE  Baseline:  Comprehensive  Engineering  Description. 
Includes  CARD  material  with  more  detail. 

^CARD:  includes  system  attributes  and 
references  programmatic  strategies 


Core  Technical  Design 
and  Physical  Description 


DoD  5000.4-M,  1992  DoD  Instruction, 
with  aspects  written  at  a  high  level 

Implementation  activities,  events,  plans  and  technical  details  that 
influence  the  acquisition  engineering  effort  and  cost/schedule 
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AEF  Example 


Threat 


CONORS 

Force 

Structure 


Programmatic 

Drivers 


i 


Operational 

Requirements 


Activity  Lead 
PMO  Team 


External 


cONOPS 


± 


System 

Requirements 


Step  4 


Technology 

Assessment 


Contractor 

Proposals 


Reference 

Design 


Risk 

Assessment 


ASE  Baseline 


( 

\ 


Loop  iterated  multiple  times  frorn^ 
program  conception  through  CDR  J 


Tightly  Coupled  Activity 

* 


Step  3 


Trade-Offs 


Technical  Description 


Performance 

Analysis 


Effectiveness 

Analysis 


j 


■  "Reference  design"  is  key  to  coherency 

■  "Tightly  coupled  activity"  permits  rapid  projection  of  performance,  cost, 
schedule,  and  risk  to  support  definition  and  refinement  of  system  requirements 
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Lessons  Learned  from  R-TOC  Program 

■  Reduction  in  Total  Ownership  Cost  (R-TOC)  was  a 

1999  DoD  initiative. 

■  2008  IDA  R-TOC  Lessons 

Learned  Memo 

-  Involve  command  cost  investment 
analysis  personnel  as  part  of  the 
program  IPT. 

-  Accurate  and  timely  data  are 
essential  to  identify  savings. 

-  Try  to  understand  the  lifecycle 
implications  when  making 
decisions. 

-  Large  savings  requires  large 
investment. 


R-TOC  Pilot  Lessons  Leai  ued/Best  Practices 


Danny  L  Reed.  Task  Leader 
Richard  Rome: 


I  DA 


Institute  for  Defense  Analyses 
4850  Mark  C  enter  Drive 
Alexandria.  Virginia  2231 1-1882 
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